Unsupervised Word Segmentation with Bi-directional Neural Language Model

We propose an unsupervised word segmentation model, in which for each unlabelled sentence sample, the learning objective is to maximize generation probability of given its all possible segmentations. Such a can be factorized into likelihood segment context recursive way. To capture both long- and short-term dependencies, we use bi-directional neural language model better extract features segment’s context. Two decoding algorithms were also developed combine from directions generate final at inference time, helps reconcile word-boundary ambiguities. Experimental results show that our context-sensitive achieved state-of-the-art different evaluation settings on various datasets Chinese, comparable result Thai.